Server-Side Encryption and Encryption At Host support

[m1kola]

Relevant installer (fork) PR: jewzaam/installer-aro#5

Which issue this PR addresses:

Fixes work item №9586080.

What this PR does / why we need it:

Addds support for:

• Server-Side Encryption
• Encryption At Host support

Test plan for issue:

Execute the following command to register the feature for your subscription (Not required in ARO dev sub):

az feature register --namespace Microsoft.Compute --name EncryptionAtHost

Set basic variables

RESOURCEGROUP=$USER-test
CLUSTER=$RESOURCEGROUP-cluster
CLUSTER_RESOURCEGROUP=$CLUSTER
CLUSTER_DOMAIN=<replace-me> # dwfcu1jm, for example. Generate your own
LOCATION=eastus

KEYVAULT_NAME=$USER-kv
KEYVAULT_KEY_NAME=$USER-disk-encryption-key
DISK_ENCRYPTION_SET_NAME=$USER-disk-encryption-set

CLUSTER_SP_NAME=$CLUSTER
AZURE_FP_CLIENT_ID=<known-value-from-sub>

Create resource group for vnet, disk encryption set and cluster object

Resource group creation:

az group create \
  --name $RESOURCEGROUP \
  --location $LOCATION

Vnet and subnets creation:

az network vnet create \
   --resource-group $RESOURCEGROUP \
   --name aro-vnet \
   --address-prefixes 10.0.0.0/22

az network vnet subnet create \
  --resource-group $RESOURCEGROUP \
  --vnet-name aro-vnet \
  --name master-subnet \
  --address-prefixes 10.0.0.0/23 \
  --service-endpoints Microsoft.ContainerRegistry

az network vnet subnet create \
  --resource-group $RESOURCEGROUP \
  --vnet-name aro-vnet \
  --name worker-subnet \
  --address-prefixes 10.0.2.0/23 \
  --service-endpoints Microsoft.ContainerRegistry

az network vnet subnet update \
  --name master-subnet \
  --resource-group $RESOURCEGROUP \
  --vnet-name aro-vnet \
  --disable-private-link-service-network-policies true

Create a cluster service principal

Make a note of appId and password.

az ad sp create-for-rbac --skip-assignment --name $CLUSTER_SP_NAME

CLUSTER_SP_ID=<appId>
CLUSTER_SP_SECRET=<password>

Grant network contributor to cluster service principal and first part service principal.

VNET_RESOURCE_ID=$(az network vnet show --resource-group $RESOURCEGROUP --name aro-vnet --query "[id]" -o tsv)

az role assignment create --assignee $CLUSTER_SP_ID --role "Network Contributor" --scope $VNET_RESOURCE_ID -o jsonc
az role assignment create --assignee $AZURE_FP_CLIENT_ID --role "Network Contributor" --scope $VNET_RESOURCE_ID -o jsonc

Create Key Vault

az keyvault create -n $KEYVAULT_NAME -g $RESOURCEGROUP -l $LOCATION --enable-purge-protection true --enable-soft-delete true

az keyvault key create --vault-name $KEYVAULT_NAME -n $KEYVAULT_KEY_NAME --protection software

Create an instance of a DiskEncryptionSet

KEYVAULT_ID=$(az keyvault show --name $KEYVAULT_NAME --query "[id]" -o tsv)

KEYVAULT_KEY_URL=$(az keyvault key show --vault-name $KEYVAULT_NAME --name $KEYVAULT_KEY_NAME --query "[key.kid]" -o tsv)

az disk-encryption-set create -n $DISK_ENCRYPTION_SET_NAME -l $LOCATION -g $RESOURCEGROUP --source-vault $KEYVAULT_ID --key-url $KEYVAULT_KEY_URL

Grant the DiskEncryptionSet resource access to the key vault:

DES_IDENTITY=$(az disk-encryption-set show -n $DISK_ENCRYPTION_SET_NAME -g $RESOURCEGROUP --query "[identity.principalId]" -o tsv)

az keyvault set-policy -n $KEYVAULT_NAME -g $RESOURCEGROUP --object-id $DES_IDENTITY --key-permissions wrapkey unwrapkey get

Grant cluster and First Party SP Reader permissions to DiskEncryptionSet

DES_RESOURCE_ID=$(az disk-encryption-set show -n $DISK_ENCRYPTION_SET_NAME -g $RESOURCEGROUP --query "[id]" -o tsv)

az role assignment create --assignee $CLUSTER_SP_ID --role Reader --scope $DES_RESOURCE_ID -o jsonc
az role assignment create --assignee $AZURE_FP_CLIENT_ID --role Reader --scope $DES_RESOURCE_ID -o jsonc

Create a dev cluster:

SUBSCRIPTION_ID=$(az account show --query "[id]" -o tsv)
MASTER_SUBNET_RESOURCE_ID=$(az network vnet subnet show -g $RESOURCEGROUP --vnet-name aro-vnet --name master-subnet --query "[id]" -o tsv)
WORKER_SUBNET_RESOURCE_ID=$(az network vnet subnet show -g $RESOURCEGROUP --vnet-name aro-vnet --name worker-subnet --query "[id]" -o tsv)

REQUEST_DATA=$(cat << EOF
{
    "location": "$LOCATION",
    "properties": {
        "clusterProfile": {
            "domain": "$CLUSTER_DOMAIN",
            "resourceGroupId": "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$CLUSTER_RESOURCEGROUP"
        },
        "servicePrincipalProfile": {
            "clientId": "$CLUSTER_SP_ID",
            "clientSecret": "$CLUSTER_SP_SECRET"
        },
        "networkProfile": {
            "podCidr": "10.128.0.0/14",
            "serviceCidr": "172.30.0.0/16",
            "sdnProvider": "OpenShiftSDN"
        },
        "masterProfile": {
            "vmSize": "Standard_D8s_v3",
            "subnetId": "$MASTER_SUBNET_RESOURCE_ID",
            "encryptionAtHost": true,
            "diskEncryptionSetId": "$DES_RESOURCE_ID"
        },
        "workerProfiles": [
            {
                "name": "worker",
                "vmSize": "Standard_D2s_v3",
                "diskSizeGB": 130,
                "subnetId": "$WORKER_SUBNET_RESOURCE_ID",
                "count": 3,
                "encryptionAtHost": true,
                "diskEncryptionSetId": "$DES_RESOURCE_ID"
            }
        ],
        "apiserverProfile": {
            "visibility": "Public"
        },
        "ingressProfiles": [
            {
                "name": "default",
                "visibility": "Public"
            }
        ]
    }
}
EOF
)


curl -k -X PUT "https://localhost:8443/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.RedHatOpenShift/openShiftClusters/$CLUSTER?api-version=2021-09-01-preview" -H 'Content-Type: application/json' -d $REQUEST_DATA

Check that all VMs have correct properties:

az vm list -g $CLUSTER_RESOURCEGROUP --query "[].{Name:name, diskEncryptionSet: storageProfile.osDisk.managedDisk.diskEncryptionSet.id, encryptionAtHost: securityProfile.encryptionAtHost}"

Note: you can run this command while bootstrap nodes still exist: they should have the same properties as set in masterProfile.

Is there any documentation that needs to be updated for this PR?

We need to (out of scope of this work item):

• Update az CLI
• Customer facing docs

[m1kola]

I'll create separate PRs for:

• Validations
• Update of default storage class (need to look more into this)

But this is already in a state where we can merge, I think. Want to keep PR small, relatively easy to review.

Note that az cli enablement and documentation are not in scope of the work item.

/azp run e2e

[mjudeikis]

Interesting short version of this :)

[mjudeikis]

Lets merge this post Arch discussion about E2E. This is something I would like to see in E2E. Maybe extend current tests to have this in?

[m1kola]

@mjudeikis while working on this I accidentaly broke all cluster creations without disk encryption set (non-priview APIs basically), but preview API was working. So I would be opposed to the idea of using preview API in the e2e at this stage.

I suggest that we do not block this PR by the fact that we do not have a way to test different versions in e2s. I'm happy to create a separate task/subtast for this and work on it.

[mjudeikis]

@m1kola I think you right. We need separate story "E2E tests with new preview api" and we will have second flavor of e2e later.
With OVN, disk encryption, kubeconfig download tests...