Skip to content

Commit

Permalink
Update AKS documentation (#514)
Browse files Browse the repository at this point in the history
* Update AKS docs
  • Loading branch information
jsaun authored Oct 17, 2022
1 parent 9e85a9e commit 13bc41f
Showing 1 changed file with 48 additions and 11 deletions.
59 changes: 48 additions & 11 deletions docs/coa-aks.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ The CoA deployer requires the user to have Helm 3 installed locally to deploy wi
### Deployment Models

- ### CoA provisioned AKS account
Add the flag "--UseAks true --UserObjectId aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" and the deployer will provision an AKS account and run its containers in AKS rather than provisioning a VM. UserObjectId is required to assign permissions for the deployer to use KeyVault which is used by CoA to store the storage account key.
Add the flag "--UseAks true" and the deployer will provision an AKS account and run its containers in AKS rather than provisioning a VM.
- ### Shared AKS account with CoA namespace
Add the flags "--UseAks true --AksClusterName {existingClusterName} --UserObjectId aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", where the user has "Contributor" or "Azure Kubernetes Service Contributor" role access to the existing AKS account, the deployer will deploy blob-csi-driver, and aad-pod-identity to the kube-system namespace, and then deploy CoA to the namespace "coa". Add the flag "--AksCoANamespace {namespace}" to override the default namespace.
Add the flags "--UseAks true --AksClusterName {existingClusterName}", where the user has "Contributor" or "Azure Kubernetes Service Contributor" role access to the existing AKS account, the deployer will deploy blob-csi-driver, and aad-pod-identity to the kube-system namespace, and then deploy CoA to the namespace "coa". Add the flag "--AksCoANamespace {namespace}" to override the default namespace.
- ### Shared AKS account without developer access.
If the user is required to use an AKS account, but does not have the required access, the deployer will produce a Helm chart that can then be installed by an admin or existing CI/CD pipeline. Add the flags "--UseAks true --ManualHelmDeployment --UserObjectId aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee". The deployer will print a postgresql command, this would typically be run on the kubernetes node to setup the cromwell user however the user will need to run this manually since the deployer won't directly access the AKS account.
If the user is required to use an AKS account, but does not have the required access, the deployer will produce a Helm chart that can then be installed by an admin or existing CI/CD pipeline. Add the flags "--UseAks true --ManualHelmDeployment". The deployer will print a postgresql command, this would typically be run on the kubernetes node to setup the cromwell user however the user will need to run this manually since the deployer won't directly access the AKS account.

- Run the deployer with supplied flags.
- Deployer will create initial resources and pause once it's time to deploy the Helm chart.
Expand All @@ -24,10 +24,7 @@ These packages will be deployed into the kube-system namespace.
- ### Blob CSI Driver - https://github.com/kubernetes-sigs/blob-csi-driver/
This is used to mount the storage account to the containers.
- ### AAD Pod Identity - https://github.com/Azure/aad-pod-identity
This is used to assigned managed identities to the containers.

### External storage accounts
Typically, in CromwellOnAzure you can add storage accounts with input data to the containers-to-mount file. For AKS, you need to modify the values.yaml file of the helm chart and redeploy. The vaules-template.yaml will have examples externalContainers and externalSasContainers.
This is used to assign managed identities to the containers.

### Logs and troubleshooting
For troubleshooting any of the CoA services, you can login directly to the pods or get logs using the kubectl program. The deployer will write a kubeconfig to the temp directory, either copy that file to ~/.kube/config for reference it manually for each command with --kubeconfig {temp-directory}/kubeconfig.txt. You can also run the command `az aks get-credentials --resource-group {coa-resource-group} --name {aks-account} --subscription {subscription-id} --file kubeconfig.txt` to get the file.
Expand All @@ -45,9 +42,49 @@ For troubleshooting any of the CoA services, you can login directly to the pods
### Updating settings and environment variables.

For VM based CoA deployments, you can ssh into the VM host, update the environment files, and restart the VM.
To update settings for AKS, you will need to redeploy the helm chart. If you still have the chart locally,
you can update the values.yaml file and redeploy with:
To update settings for AKS, you will need to redeploy the helm chart. The configuration for the helm chart is
stored in CoA default storage account /configuration/aksValues.yaml. Setting in this file such as the image
versions and external storage accounts can be updated and redeployed with the deployer update command.

`deploy-cromwell-on-azure.exe --update true --aksclustername clustername`

### External storage accounts
Typically, in CromwellOnAzure you can add storage accounts with input data to the containers-to-mount file. For AKS, the external storage accounts are configured
in the aksValues.yaml in the storage account. There are three methods for adding storage accounts
to your deployment.

1. Managed Identity
2. Storage Key

a) Plain text - "externalContainers"

b) Key Vault - "internalContainersKeyVaultAuth"
3. SAS Token

```
internalContainersMIAuth:
- accountName: storageAccount
containerName: dataset1
resourceGroup: resourceGroup1
internalContainersKeyVaultAuth:
- accountName: storageAccount
containerName: dataset1
keyVaultURL:
keyVaultSecretName:
externalContainers:
- accountName: storageAccount
accountKey: <key>
containerName: dataset1
resourceGroup: resourceGroup1
externalSasContainers:
- accountName:
sasToken:
containerName:
```

`helm upgrade cromwellonazure ./scripts/helm --kubeconfig kubeconfig.txt --namespace coa`
To add new storage accounts, please update the aksValues.yaml according to the above template and run the update.

If the original chart is lost, you can regenerate it by running the deployer again with the "--update true" and "--AksClusterName {existingClusterName}" flags.
The default storage account will be mounted using either the internalContainersMIAuth or internalContainersKeyVaultAuth depending on if the CrossSubscriptionAKSDeployment flag was used during deployment.

0 comments on commit 13bc41f

Please sign in to comment.