ARO Fips Mode Installation

[ross-bryan]

Which issue this PR addresses:

Fixes https://msazure.visualstudio.com/AzureRedHatOpenShift/_workitems/edit/10901527/

What this PR does / why we need it:

Exposes a Cluster Profile parameter named FipsValidatedModules which is sent to the installer to determine if FIPS mode is turned on for cluster.

Test plan for issue:

Added frontend tests, defaults tests, e2e tests checking database document and machineconfigs for FIPS, and manual verification process:

Setup basic vars and create network:

RESOURCEGROUP=$USER-fips
CLUSTER=$RESOURCEGROUP-cluster
CLUSTER_RESOURCEGROUP=$CLUSTER
LOCATION=eastus
AZURE_FP_CLIENT_ID=#well known value
CLUSTER_SP_NAME=$CLUSTER
CLUSTER_DOMAIN=#yourdomainhere

az group create --name $RESOURCEGROUP --location $LOCATION

az network vnet create \
   --resource-group $RESOURCEGROUP \
   --name aro-vnet \
   --address-prefixes 10.0.0.0/22

az network vnet subnet create \
  --resource-group $RESOURCEGROUP \
  --vnet-name aro-vnet \
  --name master-subnet \
  --address-prefixes 10.0.0.0/23 \
  --service-endpoints Microsoft.ContainerRegistry

az network vnet subnet create \
  --resource-group $RESOURCEGROUP \
  --vnet-name aro-vnet \
  --name worker-subnet \
  --address-prefixes 10.0.2.0/23 \
  --service-endpoints Microsoft.ContainerRegistry

az network vnet subnet update \
  --name master-subnet \
  --resource-group $RESOURCEGROUP \
  --vnet-name aro-vnet \
  --disable-private-link-service-network-policies true

Next create the SP, and note the values and assign therem

az ad sp create-for-rbac --skip-assignment --name $CLUSTER_SP_NAME
CLUSTER_SP_ID=#id
CLUSTER_SP_SECRET=#password

Give network contributor permissions on SP, and generate JSON payload with FipsEnabled in ClusterProfile

VNET_RESOURCE_ID=$(az network vnet show --resource-group $RESOURCEGROUP --name aro-vnet --query "[id]" -o tsv)
az role assignment create --assignee $CLUSTER_SP_ID --role "Network Contributor" --scope $VNET_RESOURCE_ID -o jsonc
az role assignment create --assignee $AZURE_FP_CLIENT_ID --role "Network Contributor" --scope $VNET_RESOURCE_ID -o jsonc

SUBSCRIPTION_ID=$(az account show --query "[id]" -o tsv)

MASTER_SUBNET_RESOURCE_ID=$(az network vnet subnet show -g $RESOURCEGROUP --vnet-name aro-vnet --name master-subnet --query "[id]" -o tsv)
WORKER_SUBNET_RESOURCE_ID=$(az network vnet subnet show -g $RESOURCEGROUP --vnet-name aro-vnet --name worker-subnet --query "[id]" -o tsv)

data.json:

{
    "location": "$LOCATION",
    "properties": {
        "clusterProfile": {
            "domain": "$CLUSTER_DOMAIN",
            "resourceGroupId": "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$CLUSTER_RESOURCEGROUP",
	    "fipsValidatedModules": "Enabled"
        },
        "servicePrincipalProfile": {
            "clientId": "$CLUSTER_SP_ID",
            "clientSecret": "$CLUSTER_SP_SECRET"
        },
        "networkProfile": {
            "podCidr": "10.128.0.0/14",
            "serviceCidr": "172.30.0.0/16",
            "softwareDefinedNetwork": "OpenShiftSDN"
        },
        "masterProfile": {
            "vmSize": "Standard_D8s_v3",
            "subnetId": "$MASTER_SUBNET_RESOURCE_ID",
            "encryptionAtHost": "Disabled"

        },
        "workerProfiles": [
            {
                "name": "worker",
                "vmSize": "Standard_D2s_v3",
                "diskSizeGB": 130,
                "subnetId": "$WORKER_SUBNET_RESOURCE_ID",
                "count": 3,
                "encryptionAtHost": "Disabled"

            }
        ],
        "apiserverProfile": {
            "visibility": "Public"
        },
        "ingressProfiles": [
            {
                "name": "default",
                "visibility": "Public"
            }
        ]
    }
}

Send payload to local RP to build cluster

curl -k -X PUT "https://localhost:8443/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.RedHatOpenShift/openShiftClusters/$CLUSTER?api-version=2021-09-01-preview" -H 'Content-Type: application/json' -d @./data.json

Once cluster is built, we can query for existence of FipsValidatedModules in ClusterProfile:

curl -k https://localhost:8443/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.RedHatOpenShift/openShiftClusters/$CLUSTER?api-version=2021-09-01-preview" -H 'Content-Type: application/json'

Validate adminapi as well:

curl -k "https://localhost:8443/subscriptions/225e02bc-43d0-43d1-a01a-17e584a4ef69/resourceGroups/robryan-fips/providers/Microsoft.RedHatOpenShift/openShiftClusters/robryan-fips-cluster?api-version=admin" -H 'Content-Type: application/json'

Now we can manually test that sysctl fips_enabled is true:

make admin.kubeconfig
export KUBECONFIG=admin.kubeconfig
oc debug node/#anynode
chroot /host
sh-4.4# cat /proc/sys/crypto/fips_enabled
1

Is there any documentation that needs to be updated for this PR?

Will provide document updates when this is released via az cli.

[m1kola]

Looks good so far. Left few comments which might be useful.

[m1kola]

1. Do we need to expose this to the customers?
2. If the answer to the above is "yes", then we need to update API spec in https://github.com/Azure/azure-rest-api-specs if we are still allowed to do so. Or create a new API version if not

[mjudeikis]

I suspect this will be breaking change, would be good to raise POC PR into repo @m1kola raised and confirm. Tell them not to merge, but ask a question.
This might need to be ARM feature flag for now.

[ross-bryan]

Azure/azure-rest-api-specs#16028 on my todo list to follow up on this to determine what steps are needed for breaking change approval. I know @cadenmarchese is in a similar boat as myself with L series support.

[mjudeikis]

Overall good start.
Please move CLI too separate PR as that side is not yet ready and is not a blocker to deliver this feature.

[mjudeikis]

I suspect this will be breaking change, would be good to raise POC PR into repo @m1kola raised and confirm. Tell them not to merge, but ask a question.
This might need to be ARM feature flag for now.

[mjudeikis]

Same as Admin API - no bools.

[mjudeikis]

Overall MSFT does not like abbreviation on the API but this might fly... Need POC pr for this too :)

[ross-bryan]

I am not sure what to expect, but here goes nothing Azure/azure-rest-api-specs#16028

applied changes

[nilsanderselde]

Looks good, just found one typo