CromwellOnAzure On AKS

docs/coa-aks.md

@@ -0,0 +1,53 @@
+## Cromwell on AKS Instructions and Troubleshooting
+
+### Deployment Dependencies
+The CoA deployer requires the user to have Helm 3 installed locally to deploy with AKS. Use the flag "--HelmExePath HELM_PATH" to give the deployer the path to the helm exe, if no flag is passed the deployer will assume Helm is installed with Chocolatey at "C:\\ProgramData\\chocolatey\\bin\\helm.exe".
+
+### Deployment Models
+
+- ### CoA provisioned AKS account
+    Add the flag "--UseAks true" and the deployer will provision an AKS account and run its containers in AKS rather than provisioning a VM. 
+- ### Shared AKS account with CoA namespace
+    Add the flags "--UseAks true --AksClusterName {existingClusterName}", where the user has admin access to the existing AKS account, the deployer will deploy blob-csi-driver, and aad-pod-identity to the kube-system namespace, and then deploy CoA to the namespace "coa". Add the flag "--AksCoANamespace {namespace}" to override the default namespace.
+- ### Shared AKS account without developer access. 
+    If the user is required to use an AKS account, but does not have the required access, the deployer will produce a Helm chart that can then be installed by an admin or existing CI/CD pipeline. Add the flags "--UseAks true --ManualHelmDeployment". The deployer will print a postgresql command, this would typically be run on the kubernetes node to setup the cromwell user however the user will need to run this manually since the deployer won't directly access the AKS account. 
+
+    - Run the deployer with supplied flags. 
+    - Deployer will create initial resources and pause once it's time to deploy the Helm chart.
+    - Ensure the blob-csi-driver and aad-pod-identity are installed.
+    - Install the CoA Helm chart. 
+    - Run the postgresql command to create the cromwell user. 
+    - Press enter in the deployer console to finish the deployment and run a test workflow. 
+
+### Depedent Kubernetes Packages
+These packages will be deployed into the kube-system namespace.
+- ### Blob CSI Driver - https://github.com/kubernetes-sigs/blob-csi-driver/
+    This is used to mount the storage account to the containers.
+- ### AAD Pod Identity - https://github.com/Azure/aad-pod-identity
+    This is used to assigned managed identities to the containers. 
+
+### External storage accounts
+Typically in CromwellOnAzure you can add storage accounts with input data to the containers-to-mount file. For AKS, you need to modify the values.yaml file of the helm chart and redeploy. The vaules-template.yaml will have examples externalContainers and externalSasContainers. 
+
+### Logs and troubleshooting
+For troubleshooting any of the CoA services, you can login directly to the pods or get logs using the kubectl program. The deployer will write a kubeconfig to the working directory, either copy that file to ~/.kube/config for reference it manually for each command with --kubeconfig {coa-directory}/kubeconfig.txt. You can also run the command `az aks get-credentials --resource-group {coa-resource-group} --name {aks-count} --subscription {subscription-id} --file kubeconfig.txt` to get the file.
+
+1. Get the exact name of the pods. 
+
+    `kubectl get pods --namespace coa`
+2. Get logs for the tes pod.
+
+    `kubectl logs tes-68d6dc4789-mvvwj --namespace coa`
+3. SSH to pod to troubleshoot storage or network connectivity.
+
+    `kubectl exec --namespace coa --stdin --tty tes-68d6dc4789-mvvwj -- /bin/bash`
+
+### Updating settings and environment variables.
+
+For VM based CoA deployments, you can ssh into the VM host, update the environment files, and restart the VM. 
+To update update settings for AKS, you will need to redeploy the helm chart. If you still have the chart locally, 
+you can update the values.yaml file and redeploy with:
+
+`helm upgrade cromwellonazure ./scripts/helm --kubeconfig kubeconfig.txt --namespace coa`
+
+If the original chart is lost, you can regenerate it by running the deployer again with the "--update true" and "--AksClusterName {existingClusterName}" flags. 

docs/troubleshooting-guide.md

@@ -208,7 +208,7 @@ Before deploying, you can choose to customize some input parameters to use exist
 .\deploy-cromwell-on-azure.exe --SubscriptionId <Your subscription ID> --RegionName <Your region> --MainIdentifierPrefix <Your string> --VmSize "Standard_D2_v2"
 ```
 
-Here is the summary of all configuration parameters:
+Here is the summary of common configuration parameters:
 
 Configuration   parameter | Has default | Validated | Used by update | Comment
 -- | -- | -- | -- | --
@@ -239,7 +239,28 @@ bool     Update =   false; | Y | Y | Y | Set to true if you want to update your
 bool     PrivateNetworking = false; | Y | Y | N | Available starting version 2.2. Set to true to create the host VM without public IP address. If set, VnetResourceGroupName, VnetName and SubnetName must be provided (and already exist). The deployment must be initiated from a machine that has access to that subnet.
 bool     KeepSshPortOpen =   false; | Y | Y | Y | Available starting version 3.0. Set to true if you need to keep the SSH port accessible on the host VM while deployer is not running (not recommended). 
 string   LogAnalyticsArmId | Y | N | N | Arm resource id for an exising Log Analytics workspace, workspace is used for App Insights - Not required, a workspace will be generated automatically if not provided.
-bool     ProvisionPostgreSqlOnAzure =   false; | Y | N | N | COMING SOON in version 4.0. Triggers whether to use Docker MySQL or Azure PostgreSQL when provisioning the database.
+bool     ProvisionPostgreSqlOnAzure =   false; | Y | N | N | Triggers whether to use Docker MySQL or Azure PostgreSQL when provisioning the database. Required for AKS deployment.
+bool     UseAks =   false; | Y | N | N | Uses Azure Kubernetes Service rather than a VM to run the CoA system services Cromwell/TES/TriggerService.
+string   AksClusterName | Y | Y | N | Cluster name of existing Azure Kubernetes Service cluster to use rather than provisioning a new one.
+string   AksCoANamespace = "coa" | Y | N | N | Kubernetes namespace.
+bool     ManualHelmDeployment | Y | N | N | For use if user doesn't have direct access to existing AKS cluster.
+string   HelmExePath = "C:\\ProgramData\\chocolatey\\bin\\helm.exe" | Y | N | N | Path to helm executable for AKS deployment.
+int      AksPoolSize = 2 | Y | N | N | Size of AKS node pool, two nodes are recommended for reliability, however a minimum of one can be used to save COGS. 
+bool DebugLogging = false | Y | N | N | Prints all log information.
+string PostgreSqlServerName | Y | Y | N | Name of existing postgresql server. 
+bool UsePostgreSqlSingleServer = false | Y | N | N | Use Postgresql single server rather than flexi servers, only recommended if you need to use private endpoints. 
+string KeyVaultName | Y | Y | N | Name of an existing key vault 
+
+The following are more advanced configuration parameters:
+
+Configuration   parameter | Has default | Validated | Used by update | Comment
+-- | -- | -- | -- | --
+string   VnetAddressSpace = "10.1.0.0/16" | Y | N | N | Total address space for CoA vnet.
+string   VmSubnetAddressSpace = "10.1.0.0/24" | Y | N | N | Address space for compute, VM or AKS. 
+string   MySqlSubnetAddressSpace  = "10.1.1.0/24" | Y | N | N | Address space for database.
+string   KubernetesServiceCidr = "10.1.4.0/22" | Y | N | N | Address space for kubernetes system services, must not overlap with any subnets.
+string   KubernetesDnsServiceIP = "10.1.4.10" | Y | N | N | Kubernetes DNS service IP Address.
+string   KubernetesDockerBridgeCidr = "172.17.0.1/16" | Y | N | N | Kubernetes dock bridge Cidr.
 
 ### Use a specific Cromwell version
 #### Before deploying Cromwell on Azure

src/deploy-cromwell-on-azure/Configuration.cs

@@ -11,7 +11,6 @@ namespace CromwellOnAzureDeployer
 {
     public class Configuration : UserAccessibleConfiguration
     {
-        public string PostgreSqlServerName { get; set; }
         public string PostgreSqlCromwellDatabaseName { get; set; } = "cromwell_db";
         public string PostgreSqlTesDatabaseName { get; set; } = "tes_db";
         public string PostgreSqlAdministratorLogin { get; set; } = "coa_admin";
@@ -23,11 +22,11 @@ public class Configuration : UserAccessibleConfiguration
         public string PostgreSqlSkuName { get; set; } = "Standard_B2s";
         public string PostgreSqlTier { get; set; } = "Burstable";
         public string DefaultVmSubnetName { get; set; } = "vmsubnet";
-        public string DefaultPostgreSqlSubnetName { get; set; } = "mysqlsubnet";
         public string PostgreSqlVersion { get; set; } = "11";
+        public string DefaultPostgreSqlSubnetName { get; set; } = "sqlsubnet";
         public int PostgreSqlStorageSize { get; set; } = 128;  // GiB
-        public bool? ProvisionPostgreSqlOnAzure { get; set; } // Will be accessible in 4.0 release
     }
+
     public abstract class UserAccessibleConfiguration
     {
         public string SubscriptionId { get; set; }
@@ -37,9 +36,15 @@ public abstract class UserAccessibleConfiguration
         public string VmOsName { get; set; } = "UbuntuServer";
         public string VmOsVersion { get; set; } = "18.04-LTS";
         public string VmSize { get; set; } = "Standard_D3_v2";
-        public string VnetAddressSpace { get; set; } = "10.1.0.0/16";
-        public string VmSubnetAddressSpace { get; set; } = "10.1.0.0/24";
-        public string PostgreSqlSubnetAddressSpace { get; set; } = "10.1.1.0/24";
+        public string VnetAddressSpace { get; set; } = "10.1.0.0/16"; // 10.1.0.0 - 10.1.255.255, 65536 IPs
+        // Address space for CoA services.
+        public string VmSubnetAddressSpace { get; set; } = "10.1.0.0/24"; // 10.1.0.0 - 10.1.0.255, 256 IPs
+        public string PostgreSqlSubnetAddressSpace { get; set; } = "10.1.1.0/24"; // 10.1.1.0 - 10.1.1.255, 256 IPs
+        // Address space for kubernetes system services, must not overlap with any subnet.
+        public string KubernetesServiceCidr = "10.1.4.0/22"; // 10.1.4.0 -> 10.1.7.255, 1024 IPs
+        public string KubernetesDnsServiceIP = "10.1.4.10";
+        public string KubernetesDockerBridgeCidr = "172.17.0.1/16"; // 172.17.0.0 - 172.17.255.255, 65536 IPs
+
         public string VmUsername { get; set; } = "vmadmin";
         public string VmPassword { get; set; }
         public string ResourceGroupName { get; set; }
@@ -50,6 +55,12 @@ public abstract class UserAccessibleConfiguration
         public string LogAnalyticsArmId { get; set; }
         public string ApplicationInsightsAccountName { get; set; }
         public string VmName { get; set; }
+        public bool UseAks { get; set; }
+        public string AksClusterName { get; set; }
+        public string AksCoANamespace { get; set; } = "coa";
+        public bool ManualHelmDeployment { get; set; }
+        public string HelmExePath { get; set; } = "C:\\ProgramData\\chocolatey\\bin\\helm.exe";
+        public int AksPoolSize { get; set; } = 2;
         public bool Silent { get; set; }
         public bool DeleteResourceGroupOnFailure { get; set; }
         public string CromwellVersion { get; set; }
@@ -72,6 +83,13 @@ public abstract class UserAccessibleConfiguration
         public string BlobxferImageName { get; set; } = null;
         public bool? DisableBatchNodesPublicIpAddress { get; set; } = null;
         public bool? KeepSshPortOpen { get; set; } = null;
+        public bool DebugLogging { get; set; } = false;
+        public bool? ProvisionPostgreSqlOnAzure { get; set; } = null;
+        public string PostgreSqlServerName { get; set; }
+        public bool UsePostgreSqlSingleServer { get; set; } = false;
+        public string KeyVaultName { get; set; }
+        // Temporary workaround until I can get Azure Graph RBAC client working. 
+        public string UserObjectId { get; set; }
 
         public static Configuration BuildConfiguration(string[] args)
         {